fix(ipc): correct skip_field handling for V4 Union#9829
Merged
Conversation
Contributor
Author
|
cc @alamb |
alamb
approved these changes
Apr 27, 2026
| } | ||
| Union(fields, mode) => { | ||
| self.skip_buffer(); // Nulls | ||
| if self.version < MetadataVersion::V5 { |
Contributor
There was a problem hiding this comment.
Rich-T-kid
pushed a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Jun 2, 2026
# Which issue does this PR close? - Closes apache#9828 . # Rationale for this change Currently, `skip_field` does not correctly handle the buffer layout of `Union` types for V4 IPC. In V4: * `Union` includes a null buffer + type_ids (+ offsets for dense) In V5: * `Union` has no null buffer, only type_ids (+ offsets for dense) `create_array` correctly handles this difference using a version check. However, `skip_field` always assumes a null buffer and does not skip `type_ids`, leading to buffer misalignment when skipping a `Union` column in V4. This can cause incorrect decoding or runtime errors for projected columns. # What changes are included in this PR? * Updated `skip_field` in `arrow-ipc/src/reader.rs` to: * conditionally skip the null buffer only for V4 * explicitly skip the `type_ids` buffer * correctly handle dense vs sparse offsets * Aligns `skip_field` behavior with `create_array` and actual IPC layout # Are these changes tested? Yes. * Added test: `test_projection_skip_union_v4` * The test: * writes IPC data using V4 metadata * includes a `Union` column followed by an `Int32` column * reads only the second column (skipping the `Union`) * verifies the output matches expected values * The test fails before the fix and passes after * All existing `arrow-ipc` tests pass (`cargo test -p arrow-ipc --lib`) # Are there any user-facing changes? No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
skip_field#9828 .Rationale for this change
Currently,
skip_fielddoes not correctly handle the buffer layout ofUniontypes for V4 IPC.In V4:
Unionincludes a null buffer + type_ids (+ offsets for dense)In V5:
Unionhas no null buffer, only type_ids (+ offsets for dense)create_arraycorrectly handles this difference using a version check. However,skip_fieldalways assumes a null buffer and does not skiptype_ids, leading to buffer misalignment when skipping aUnioncolumn in V4.This can cause incorrect decoding or runtime errors for projected columns.
What changes are included in this PR?
Updated
skip_fieldinarrow-ipc/src/reader.rsto:type_idsbufferAligns
skip_fieldbehavior withcreate_arrayand actual IPC layoutAre these changes tested?
Yes.
Added test:
test_projection_skip_union_v4The test:
Unioncolumn followed by anInt32columnUnion)The test fails before the fix and passes after
All existing
arrow-ipctests pass (cargo test -p arrow-ipc --lib)Are there any user-facing changes?
No.